Skip to content

feat(aws-lambda-microvm): refresh identity on run hook - #19781

Closed
litianningdatadog wants to merge 4 commits into
tianning.li/3-runtime-id-refresh-consumersfrom
tianning.li/4-microvm-run-hook-activation
Closed

feat(aws-lambda-microvm): refresh identity on run hook#19781
litianningdatadog wants to merge 4 commits into
tianning.li/3-runtime-id-refresh-consumersfrom
tianning.li/4-microvm-run-hook-activation

Conversation

@litianningdatadog

@litianningdatadog litianningdatadog commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

AWS Lambda MicroVM instances created from the same image start with the same in-memory runtime id and Remote Config client id. The platform POST /aws/lambda-microvms/runtime/v1/run request is the earliest shared signal that a restored instance is becoming active.

This registers the web request listener only inside MicroVM images, matches the fixed /run hook, and refreshes identity once per process. The once-only guard handles stacked web layers observing the same platform request.

Testing

Added hook matcher coverage for exact match, ignored requests, listener registration outside MicroVMs, root-span ordering, and concurrent observations. Updated web event tests to use the real MicroVM hook path.

Risks

Medium. This is the PR that activates identity refresh from request dispatch, but only when AWS_LAMBDA_MICROVM_IMAGE_ARN is present and only for the fixed platform hook.

Additional Notes

Stacked on #19780.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.riot/requirements/1362718.txt                                          @DataDog/apm-python
.riot/requirements/13ed954.txt                                          @DataDog/apm-python
.riot/requirements/1638d0f.txt                                          @DataDog/apm-python
.riot/requirements/180047f.txt                                          @DataDog/apm-python
.riot/requirements/3cb8c3c.txt                                          @DataDog/apm-python
.riot/requirements/43423a3.txt                                          @DataDog/apm-python
ddtrace/_monkey.py                                                      @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/appsec/_asm_request_context.py                                  @DataDog/asm-python
ddtrace/appsec/_remoteconfiguration.py                                  @DataDog/asm-python
ddtrace/contrib/internal/asgi/middleware.py                             @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/bottle/patch.py                                @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/bottle/trace.py                                @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/cherrypy/patch.py                              @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/django/response.py                             @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/falcon/middleware.py                           @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/flask/patch.py                                 @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/http_server/__init__.py                        @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/http_server/patch.py                           @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/molten/patch.py                                @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/pyramid/trace.py                               @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/sanic/patch.py                                 @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/tornado/handlers.py                            @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/debugging/_probe/status.py                                      @DataDog/debugger-python
ddtrace/internal/core/__init__.py                                       @DataDog/apm-core-python
ddtrace/internal/core/crashtracking.py                                  @DataDog/profiling-python @DataDog/apm-core-python
ddtrace/internal/native/_native.pyi                                     @DataDog/apm-core-python
ddtrace/internal/remoteconfig/client.py                                 @DataDog/remote-config @DataDog/apm-core-python
ddtrace/internal/runtime/__init__.py                                    @DataDog/apm-sdk-capabilities-python
ddtrace/internal/runtime/runtime_metrics.py                             @DataDog/apm-sdk-capabilities-python
ddtrace/internal/settings/_supported_configurations.py                  @DataDog/apm-python
ddtrace/internal/settings/asm.py                                        @DataDog/asm-python
ddtrace/internal/symbol_db/symbols.py                                   @DataDog/debugger-python
ddtrace/internal/telemetry/writer.py                                    @DataDog/apm-python
ddtrace/internal/writer/writer.py                                       @DataDog/apm-core-python
docs/integrations.rst                                                   @DataDog/python-guild
releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml  @DataDog/apm-python
riotfile.py                                                             @DataDog/apm-python
scripts/integration_registry/registry.yaml                              @DataDog/apm-idm-python
src/native/crashtracker.rs                                              @DataDog/profiling-python @DataDog/apm-core-python
src/native/lib.rs                                                       @DataDog/apm-core-python
supported-configurations.json                                           @DataDog/apm-python
tests/appsec/appsec/test_asm_request_context.py                         @DataDog/asm-python
tests/appsec/appsec/test_remoteconfiguration.py                         @DataDog/asm-python
tests/contrib/asgi/test_microvm_identity_refresh.py                     @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/bottle/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/cherrypy/test_microvm_identity_refresh.py                 @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/django/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/falcon/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/flask/test_microvm_identity_refresh.py                    @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/http_server/__init__.py                                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/http_server/test_http_server_patch.py                     @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/http_server/test_microvm_identity_refresh.py              @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/molten/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/pyramid/test_microvm_identity_refresh.py                  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/sanic/test_sanic.py                                       @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/suitespec.yml                                             @DataDog/python-guild
tests/contrib/tornado/test_microvm_identity_refresh.py                  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/crashtracker/test_crashtracker.py                                 @DataDog/profiling-python
tests/internal/remoteconfig/test_remoteconfig_native.py                 @DataDog/remote-config @DataDog/apm-core-python
tests/internal/symbol_db/test_symbols.py                                @DataDog/debugger-python
tests/runtime/test_runtime_metrics_api.py                               @DataDog/apm-sdk-capabilities-python
tests/telemetry/test_writer.py                                          @DataDog/apm-python
tests/tracer/runtime/test_runtime_id.py                                 @DataDog/apm-sdk-capabilities-python
tests/tracer/test_writer.py                                             @DataDog/apm-sdk-capabilities-python

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pipelines  Tests

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 12 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64, 1] — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v126532182-233089d-musllinux_1_2_aarch64, 1] — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64] — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

View all 12 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 621c074 | Docs | View more details | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Dependency direction analysis

📈 Existing violations got worse

10 pre-existing violation(s) increased in severity (e.g. their target became more depended-on, or got pulled into an import cycle), though the edge itself isn't new:

Show violations that got worse (showing 5 of 10 highest severity) 
ddtrace.internal.remoteconfig.client -×-> ddtrace.internal.runtime  (internal-core -> product:runtime, score=15, +1 vs base)
ddtrace.internal.symbol_db.symbols -×-> ddtrace.internal.runtime  (internal-core -> product:runtime, score=15, +1 vs base)
ddtrace.contrib.internal.subprocess.patch -×-> ddtrace.internal.runtime  (contrib -> product:runtime, score=14, +1 vs base)
ddtrace.internal.symbol_db.remoteconfig -×-> ddtrace.internal.runtime  (internal-core -> product:runtime, score=15, +1 vs base)
ddtrace.debugging._probe.status -×-> ddtrace.internal.runtime  (product:debugging -> product:runtime, score=13, +1 vs base)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

⚠️ Existing dependency direction violations

There are 254 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 254 highest severity) 
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.pydantic_ai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._integrations.langgraph -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._integrations.litellm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._integrations.langchain -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@pr-commenter

pr-commenter Bot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-20 11:54:47

Comparing candidate commit 621c074 in PR branch tianning.li/4-microvm-run-hook-activation with baseline commit c9ede47 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 4 performance regressions! Performance is the same for 382 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+2.820µs; +3.004µs] or [+13.074%; +13.925%]

scenario:span-start

  • 🟥 execution_time [+1.434ms; +1.614ms] or [+9.495%; +10.683%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+369.793ns; +416.833ns] or [+14.012%; +15.794%]

scenario:tracer-small

  • 🟥 execution_time [+30.437µs; +33.258µs] or [+8.986%; +9.818%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-691.623ns; +771.989ns] or [-6.332%; +7.068%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-23.503ns; +42.737ns] or [-3.870%; +7.037%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1839.113ns; +1422.673ns] or [-10.814%; +8.365%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1090.920ns; +1411.073ns] or [-8.402%; +10.868%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-333.773ns; +313.379ns] or [-9.095%; +8.539%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-289.760ns; +215.637ns] or [-9.904%; +7.371%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-63.436ns; +88.110ns] or [-5.497%; +7.634%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-3920.982ns; +4168.575ns] or [-9.560%; +10.164%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-761.786ns; +791.162ns] or [-9.338%; +9.698%]

@litianningdatadog
litianningdatadog requested a lite review from Copilot August 20, 2026 05:23
Copilot AI reviewed Aug 20, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR activates “identity refresh” for AWS Lambda MicroVM restores by hooking the earliest shared platform signal (POST /aws/lambda-microvms/runtime/v1/run) into the existing core.WEB_REQUEST_STARTING event, ensuring runtime- and identity-bound components get refreshed once per process in MicroVM environments.

Changes:

  • Add MicroVM /run hook matching + once-per-process identity refresh registration to ddtrace.internal.runtime.
  • Expand runtime-id tests to cover exact hook matching, ordering (pre-root-span), and concurrency behavior.
  • Update web framework tests to use the real MicroVM hook path constant and add a release note.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
ddtrace/internal/runtime/init.py Registers a MicroVM-only WEB_REQUEST_STARTING listener to refresh identity once when the exact /run hook is observed.
tests/tracer/runtime/test_runtime_id.py Adds subprocess tests for hook matching, ordering vs root span creation, concurrency safety, and non-MicroVM no-op behavior.
tests/contrib/asgi/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in ASGI middleware tests.
tests/contrib/bottle/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Bottle integration tests.
tests/contrib/cherrypy/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in CherryPy integration tests.
tests/contrib/django/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Django integration tests.
tests/contrib/falcon/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Falcon integration tests.
tests/contrib/flask/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions (including WSGI environ) in Flask integration tests.
tests/contrib/http_server/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in http.server integration tests.
tests/contrib/molten/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Molten integration tests.
tests/contrib/pyramid/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Pyramid integration tests (including tracing-disabled path).
tests/contrib/sanic/test_sanic.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Sanic integration tests.
tests/contrib/tornado/test_microvm_identity_refresh.py Uses MICROVM_RUN_HOOK_PATH for request-start dispatch assertions in Tornado integration tests.
releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml Documents MicroVM-only stable identifier regeneration behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread ddtrace/internal/runtime/__init__.py
Comment on lines +1 to 7
import threading
import typing as t
import uuid
import weakref

from ddtrace.internal import core
from ddtrace.internal import forksafe
@litianningdatadog
feat(runtime): add explicit identity refresh
2fa8088 
Some runtimes can restore a process from a snapshot without creating a real fork. Reusing the fork hook there would give the process a fresh runtime id, but it would also record fake parent and ancestor lineage.

Add a direct refresh_identity() path for that case. Runtime-id subscribers are weakly held and isolated from each other so long-lived components can rebuild cached identity state without leaking instances or breaking the refresh caller.
@litianningdatadog
feat(web): emit request-starting event
2df658b 
The MicroVM /run hook needs to be observed before a web root span reads process identity, but Python has no single HTTP server substrate across WSGI, ASGI, and framework integrations.

Introduce a shared core event that web integrations emit once method and path are available, before root span creation. The event is deliberately generic here; it does not know about MicroVMs or refresh runtime ids yet.
@litianningdatadog
litianningdatadog force-pushed the tianning.li/3-runtime-id-refresh-consumers branch from b9a5154 to c992c42 Compare August 20, 2026 11:08
@litianningdatadog
fix(runtime): refresh identity-bound consumers
d424d55 
Rotating the runtime id is not enough by itself. Several long-lived components bake runtime or Remote Config client identity into native clients, workers, upload metadata, or tag caches.

Have those components subscribe to explicit identity refreshes and rebuild only the state that captures those ids. Fork-specific cleanup remains on the existing fork hooks so this path does not drop buffers or invent fork lineage.
@litianningdatadog
litianningdatadog force-pushed the tianning.li/3-runtime-id-refresh-consumers branch from c992c42 to d424d55 Compare August 20, 2026 11:22
@litianningdatadog
litianningdatadog force-pushed the tianning.li/4-microvm-run-hook-activation branch from 03c4230 to 55f826c Compare August 20, 2026 11:24
@litianningdatadog
feat(aws-lambda-microvm): refresh identity on run hook
621c074 
AWS Lambda MicroVM instances restored from the same image start with the same in-memory runtime id and Remote Config client id. The platform's /run lifecycle request is the earliest common signal that a restored instance is becoming active.

Register the web request listener only inside MicroVM images, match the fixed POST /run hook, and refresh identity once per process. Stacked web layers can observe the same request without rotating identity more than once.
@litianningdatadog
litianningdatadog force-pushed the tianning.li/4-microvm-run-hook-activation branch from 55f826c to 621c074 Compare August 20, 2026 11:30
@litianningdatadog litianningdatadog added the aws-microvm Work related to AWS MicroVM onboarding label Aug 21, 2026
@litianningdatadog
litianningdatadog force-pushed the tianning.li/3-runtime-id-refresh-consumers branch 2 times, most recently from 230002b to 8faa264 Compare August 23, 2026 21:26
@litianningdatadog litianningdatadog mentioned this pull request Aug 23, 2026
Draft
@litianningdatadog

litianningdatadog commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Closed as we switched to smaller PRs with vertical changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@brettlangdon brettlangdon Awaiting requested review from brettlangdon

Assignees

No one assigned

Labels

aws-microvm Work related to AWS MicroVM onboarding

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@litianningdatadog